A Comparison of Techniques for Selecting Text Collections

Techniques for evaluating queries against a distributed text document database allow uniform access to separate collections in the database. One such technique is to first choose a subset of collections, via a selection index. The index captures information about each collection such as terms occurring in documents, term statistics, and collection statistics. A possible implementation of such an index is a lexicon, which maintains a complete list of terms in the database. Another approach is to partially index the database by extracting fewer terms but maintaining some information about each document. In this paper we explore three collection-ranking techniques, two based on lexicons and the other based on partial document indexes. Our experiments show that in most cases the lexicon approaches outperform the partial index approach.